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Encouraging  Excellence 
and  Rewarding  Success 

in  Alberta’s  public  adult  learning  system 

A Proposal  for  implementing  a Performance  Funding  Envelope 


This  document  is  the  latest  in  a series  of  reports  and  correspondence  from  Advanced 
Education  and  Career  Development  about  the  development  of  a new  funding  mechanism 
for  Alberta’s  post-secondary  education  system.  This  document: 

• reports  on  the  status  of  the  various  funding  envelopes  that  have  been  established  as 
part  of  the  performance-based  funding  mechanism;  and 

• presents  a proposal  for  the  implementation  of  a Performance  Envelope  to  encourage 
and  reward  excellence  in  Alberta’ s public  adult  learning  system. 


Background 

In  October  1994,  following  extensive  consultation,  Jack  W.  Ady,  Minister  of  Advanced 
Education  and  Career  Development,  announced  a new  policy  framework  to  meet  the 
learning  needs  of  adult  Albertans. 

One  strategy  in  New  Directions  for  Adult  Learning  in  Alberta  related  to  the  provision  of 
funding  to  institutions  through  a new  mechanism  that  rewards  performance  in  providing 
accessibility,  quality  and  relevance  to  learners  at  the  lowest  possible  cost. 

To  develop  such  a funding  mechanism,  we  researched  other  jurisdictions  and  consulted 
with  Alberta’s  post-secondary  institutions  and  with  organizations  associated  with  post- 
secondary learning  in  the  province.  Comments  were  invited  on  a number  of  discussion 
papers  prepared  by  the  department,  which  outlined  proposals  for  a funding  mechanism. 

Based  on  the  feedback  received,  a new  funding  mechanism  was  developed,  consisting  of 
two  main  parts: 

1.  A general  operations  grant  to  fund  program  delivery,  administration  and  general 
infrastructure  renewal  requirements  within  each  institution’s  mandate.  The  grant 
will  be  tied  to  the  department’s  business  plan  and  institutional  performance. 

2.  Performance-driven  “envelopes”  that  target  funding 

a)  to  reward  performance  in  meeting  Alberta’s  goals  for  adult  learning,  and 

b)  assist  and  act  as  incentives  for  the  adult  learning  system  to  make  changes 
in  support  of  system  objectives. 
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The  diagram  below  illustrates  the  funding  mechanism. 


Funding  to  meet 
system  objectives 


Current  Level  of  Funding 


Funding  influenced 
by  performance 


Current  status  of  the  funding  envelopes 

The  Accessibility  Envelope  was  conceived  as  a successor  to  the  Access  Fund.  In  1994- 
95  the  Access  Fund  was  created  to  increase  students’  access  to  learning  opportunities  by 
adding,  over  three  years,  10,000  new  student  places  at  a cost  of  $47  million.  By  1996-97, 
plans  were  in  place  to  establish  more  than  10,000  places  at  an  ongoing  cost  of  $26 
million.  Owing  to  the  successful  results  of  the  Access  Fund,  the  fund  was  suspended  and 
the  implementation  of  an  Accessibility  Envelope  has  been  postponed  for  three  years.  The 
Accessibility  Envelope  will  be  activated  when  enrolment  demands  warrant  its 
implementation. 

The  Infrastructure  Renewal  Envelope  will  fund,  on  a matching  basis,  investment  in 
approved  equipment  and  long-term  projects  such  as  the  upgrading  and  replacement  of 
equipment  and  facilities.  No  new  construction  is  supported  in  the  department’s  current 
business  plan.  Beginning  in  1996-97,  the  department  secured  $5  million  in  new  funding, 
reallocated  $3  million  from  the  Access  Fund  (for  1996-97  only)  and  secured  $5  million 
from  Lotteries  (for  1996-98)  to  support  updating  of  equipment  used  for  learning.  Overall, 
$23  million  will  be  provided  over  1996-99.  Grants  will  be  on  a one-time,  specific  basis 
and  will  match  an  institution's  income  from  other  sources  such  as  endowments  and  the 
private  sector. 
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The  Learning  Enhancement  Envelope  encourages  innovation  to  enhance  the  quality 
and  availability  of  adult  learning  in  Alberta.  This  will  be  achieved  through  integration  of 
technology  to  the  learning-teaching  process,  both  on  and  off  campus.  Beginning  in  the 
1996-97  fiscal  year,  the  department  will  reallocate  $10  million  annually  from  the  Access 
Fund  for  a three  year  period. 

The  Research  Excellence  Envelope  was  implemented  to  support  research  excellence. 
Beginning  in  the  1996-97  fiscal  year,  $2  million  will  be  reallocated  annually  from  the 
Access  Fund  for  three  years. 

The  Performance  Envelope  is  intended  to  reward  performance  and  promote  excellence 
in  our  adult  learning  system  by  linking  funding  to  performance.  In  June  1996,  the 
Minister  announced  the  planned  implementation  of  this  envelope.  The  Performance 
Envelope  will  stimulate  post-secondary  institutions’  progress  towards  achieving  the 
system-wide  goals  of  accessibility,  responsiveness,  affordability,  research  excellence  and 
accountability.  It  will  be  the  main  instrument  of  adjustment  for  the  general  operations 
grant  to  the  institutions.  Alberta  will  be  the  first  jurisdiction  in  Canada  to  implement 
such  a mechanism. 

Beginning  in  1997-98,  $15  million  in  new  funding  will  be  provided  through  the  envelope. 
A further  $15  million  will  be  distributed  in  1998-99.  Institutions  will  also  contribute  one 
half  of  one  percent  of  their  operations  grants  in  1997-98  and  1998-99.  It  is  proposed  that 
each  institution  would  receive  a minimum  award  of  1.5  percent  for  1997-98  and  1998-99. 


Implementation  of  a Performance  Envelope 

Funding  from  the  Performance  Envelope  to  an  individual  institution  will  be  a function  of 
each  institution's  progress  towards  achieving  the  goals  for  the  adult  learning  system: 
accessibility,  responsiveness,  affordability,  research  excellence  and  accountability. 
Progress  will  be  measured  by  selected  key  performance  indicator  results  as  compared  to 
benchmarks  of  high  performance. 

A proposed  method  for  determining  the  performance  rating  calculation  using  key 
performance  indicators  is  presented  in  this  paper. 

Key  Performance  Indicators 

Development  of  KPIs  has  been  a consultative  process  among  institutions  and  the 
department.  In  October  1993,  the  department  published  Accountability:  Expectations  of 
the  Public  Post-Secondary  System , as  a discussion  paper  to  initiate  development  of  a set 
of  KPIs.  The  paper  proposed  three  steps: 

(1)  agreement  on  areas  to  measure  and  the  indicator  set; 

(2)  develop  the  definitions  and  methodologies;  and, 

(3)  collect  the  data  and  publish  the  results. 


The  Information  Reporting  Exchange  Project  (IREP)  was  initiated  to  establish  data 
definitions  for  reporting  KPIs.  In  the  fall  of  1995,  a pilot  phase  of  data  collection  was 
begun.  KPI  reporting  manuals  were  distributed  in  May  1996  to  begin  the  second  round  of 
data  collection  to  ensure  comparability. 

Currently,  the  Office  of  the  Auditor  General,  at  the  request  of  the  department  and  post- 
secondary institutions,  is  completing  a “specified  procedures  audit”  of  selected  KPIs 
across  six  institutions.  Feedback  will  be  consolidated  and  distributed  to  institutions  in 
January.  A first  annual  report  by  the  Minister  on  the  status  and  health  of  Alberta’s  post- 
secondary system,  as  described  by  KPIs,  is  planned  for  early  in  the  1997-98  fiscal  period. 

Benchmarks 

To  provide  the  proper  context  in  which  KPIs  can  be  compared,  benchmarks  of  excellence 
and  high  performance  needed  to  be  established.  Benchmarks  are  defined  as  best  known 
business  practices  indicating  superior  performance.  Benchmarks  are  adopted  as  targets 
for  optimal  organization  performance. 

Advanced  Education  and  Career  Development  published  a discussion  paper.  Benchmarks 
for  Alberta’s  Post-secondary  Education  System,  in  July  1996.  The  document  presented 
some  of  the  underlying  principles  and  challenges  that  need  to  be  considered  in  the 
development  of  benchmarks.  In  general,  institutions  supported  the  following  principles: 

• Performance  is  a combination  of  performance  against  self,  comparison  against  others, 
and  comparison  relative  to  a standard  of  excellence. 

• Institutions  want  to  be  benchmarked  against  superior  performance. 

• Where  relevant,  institutions  would  be  differentiated  based  on  size,  mandate  and 
geography. 

• Determining  the  level  of  comparison  for  a particular  indicator  would  depend  on 
whether  the  indicator  is  related  to  program. 

• In  general,  with  the  exception  of  the  research  indicators,  benchmarking  would  first 
occur  based  on  Alberta  experience,  due  to  the  lack  of  external  data. 

• Determining  whether  an  indicator  would  consist  of  year-to-year,  multi-year  averaging 
or  be  corridor-based  would  depend  on  the  statistical  properties  of  the  indicator. 


The  performance  rating  calculation 

Each  institution  would  be  given  a performance  rating:  1,  2,  or  3 for  each  indicator  as 
well  as  an  overall  performance  rating.  For  each  indicator,  institutions  with  the  highest 
rating  would  be  given  a “1”,  those  with  the  second  highest  rating,  a “2”,  and  those  with 
the  third  highest  rating,  a “3”.  The  rating  would  be  directly  linked  to  a performance 
award. 

The  performance  rating  calculation  proposed  and  presented  on  page  1 1 reflects  the 
concepts  of  previous  discussion  papers.  However,  the  Performance  Envelope  will  now  be 
subdivided  into  two  components:  learning  and  research.  All  institutions  will  be  eligible 


for  an  award  from  the  learning  pool.  Progress  toward  research  excellence  will  be 
measured  for  universities  only. 

The  indicators 

The  performance  rating  calculation  on  page  1 1 shows  the  indicators  we  are  proposing  be 
used  for  1997-98  and  1998-99  as  well  as  the  weighting  assigned  to  each.  Immediately 
following  is  a chart  that  lists  the  specific  years  of  data  required  for  each  indicator.  The 
intention  is  to  fix  the  calculation  for  1997-98  and  1998-99,  in  the  interests  of 
predictability  and  stability.  Fine-tuning  may  take  place,  but  the  indicators  would  not 
change.  For  example,  one  area  that  requires  further  work  is  activity  associated  with 
brokeraged  programs  and  institutional  collaboration. 

Throughout  the  consultations,  we  heard  that  the  measurements  relative  to  benchmarks 
should  take  into  consideration  differences  among  institutions  owing  to  size,  geography 
and  program  mandate;  that  it  should  be  simple  and  easy  to  understand;  and  that  corridors 
and  multi-year  data  should  be  used  when  possible.  As  well,  we  heard  it  is  necessary  to 
acknowledge  an  institution’s  current  standing  or  performance. as  well  as  improvement 
from  one  year  to  the  next. 

The  following  section  provides  an  overview  of  the  indicators  used  for  each  goal.  In 
choosing  the  indicators,  we  used  several  criteria  including  simplicity,  explanatory  power 
in  measuring  progress  toward  goals  and  objectives,  reliability,  understandability  and 
comparability. 


The  learning  component  of  the  Performance  Envelope 

Accessibility 

Goal  The  system  will  increase  access  for  motivated  Albertans  to  a diverse  range  of 
quality  learning  opportunities. 

Two  indicators  would  be  used: 

• Full-time  equivalent  (FTE)  credit  enrolment;  and 

• completion  rate. 

Credit  FTE  has  been  used  as  a measure  of  enrolment  across  Canada  for  some  time. 
Definitions  are  sufficiently  refined  for  comparability.  It  measures  institutions’  relative 
effort  to  provide  access  to  credentialed  programs. 

While  access  to  learning  is  important,  it  is  also  important  that  students  graduate  within  a 
reasonable  time  period.  No  standard  of  achievement  for  completion  rate  has  been  set. 
Public  policy  preferences  are  not  clear:  some  argue  that  open  access  and  completion  of 
some  portion  of  a diploma  or  degree  program  is  better  than  no  exposure  at  all;  others  take 
the  view  that,  to  be  meaningful,  access  to  learning  must  lead  to  completion  of  a program 
of  study.  Nonetheless,  given  entrance  requirements,  it  is  reasonable  to  expect  institutions 
to  improve  continuously  their  completion  rate,  while  maintaining  program  quality. 
Completion  rate  is  given  about  one  quarter  of  the  weight  of  Credit  FTE;  institutions  are 


therefore  not  encouraged  to  tighten  entrance  requirements  at  the  expense  of  access. 
Discussion  is  required  as  to  whether  this  indicator  has  been  sufficiently  developed  to  be 
used  at  this  stage.  Options  may  be  necessary. 

Eventually  rolling  three-year  averages  would  be  used  for  both  Credit  FTE  and  completion 
rate  to  minimize  the  effect  of  year-to-year-fluctuations. 


Responsiveness 

Goal  The  system  will  increase  its  responsiveness  to  the  needs  of  individual  learners 
and  to  the  social,  cultural  and  economic  needs  of  the  province. 

Two  indicators  would  be  used: 

• student  satisfaction;  and 

• employment. 

Both  measures  are  the  product  of  instruction.  While  there  is  a general  expectation  that 
people  will  gain  social  and  cultural  benefits  from  their  adult  learning,  especially  from 
university  level  programs,  most  Albertans  expect  that,  at  a minimum,  their  learning  will 
provide  them  with  skills  and  knowledge  to  participate  in  the  economy.  The  graduate 
employment  rate  is  a proxy  measure  of  institutions’  success  in  meeting  this  expectation. 
The  social  and  cultural  outcomes,  while  important,  have  not  been  clearly  articulated.  The 
general  satisfaction  of  graduates  is  intended  to  capture  the  degree  of  success  that  they  feel 
institutions  have  achieved  relative  to  these  expectations. 

Affordability 

Goal  The  system  will  provide  quality  learning  opportunities  to  the  greatest  number  of 
Albertans  at  the  lowest  possible  cost. 

Two  indicators  would  be  used: 

• cost  per  full-time  equivalent  student  (Cost/FTE);  and 

• instructional  expenditures  as  a percent  of  total  operations  expenditures. 

The  first  is  a measure  of  efficiency  of  program  delivery.  It  is  the  cost  that  must  be 
supported  by  taxpayer  grants,  tuition  and  other  revenue  sources.  The  second  indicator  is 
intended  to  capture  the  expectation  that  resources  will  be  allocated,  first  and  foremost,  to 
the  learning  function. 

The  indicators  would  eventually  encompass  three-year  rolling  averages  to  minimize  the 
effect  of  year-to-year  fluctuations. 


Accountability 

Goal  The  system  will  increase  its  accountability  to  Albertans  for  the  results  of  publicly 
funded  learning  opportunities. 

Two  indicators  would  be  used.  Points  would  be  awarded  if  an  institution  completes  and 
submits  to  the  department  its: 

• business  plan;  and 

• KPI  submission. 


The  research  component  of  the  Performance  Envelope 

Only  universities  have  a research  mandate.  Indicators  for  the  research  component  will  be 
used  for  comparison  among  the  three  campus-based  universities. 

Research  excellence 

Goal  The  system,  through  its  universities,  will  pursue  research  excellence  to  increase 
access  to  and  development  of  new  knowledge. 

Three  indicators  would  be  used: 

• council  monetary  awards  per  full-time  faculty  member; 

• citation  impact;  and 

• level  of  community  or  industry  research  funding. 

A portion  of  the  Performance  Envelope  funds  derived  from  university  operations  grants 
will  be  assigned  to  a research  excellence  pool.  Ideally,  university  operations  grants  would 
be  subdivided  into  the  funds  used  to  support  the  research  mission  and  those  used  to 
support  the  learning  mission.  As  this  data  is  not  available,  a reasonable  proxy  was 
sought.  Among  these  proxies  are  calculations  of,  for  example,  the  ratio  of  sponsored 
research  expenditures  to  total  expenditures  or  calculations  of  ‘research  intensity’  defined 
as  sponsored  research  revenue  as  a proportion  of  the  provincial  operations  grant.  We 
have  chosen  the  latter.  This  yields  a figure  of  20  percent  for  the  University  of  Alberta  and 
The  University  of  Calgary  and  8 percent  for  The  University  of  Lethbridge.  These 
percentages  would  be  applied  to  the  average  dollars  attributed  to  each  university  in  the 
Performance  Envelope,  which  is,  on  average,  2.5  percent  of  operations  grants  for  each 
university.  Effectively,  of  the  $15  million  allocated  to  the  Performance  Envelope,  for 
each  fiscal  year  of  1997-98  and  1998-99,  $13.4  million  would  be  contained  in  the 
learning  component  of  the  envelope  and  $1.6  in  the  research  component. 

Appendix  I details  further  information  about  the  indicators  used  in  the  performance  rating 
calculation.  Appendix  II  shows  in  detail  how  benchmarks  would  be  set. 


General  notes  about  the  performance  rating  calculation 

Weighting 

An  important  aspect  of  the  rating  is  the  relative  importance  assigned  to  each  of  the  goals, 
indicated  by  the  number  of  points  proposed.  Each  goal  has  two  indicators,  one  of  which  is 
usually  weighted  more  heavily  as  it  is  a lead  indicator  and  a more  representative  indicator 
of  the  goal.  The  goal  of  responsiveness  is  weighted  most  heavily.  Learning  programs  are 
of  little  value  unless  they  respond  to  the  economic  and  social  needs  of  people  and  their 
communities  regardless  of  enrolment  levels.  It  is  equally  true  that  quality,  as  measured  by 
high  levels  of  access  and  responsiveness,  is  more  highly  valued  than  efficiency.  Low  cost 
programs  are  not  a desired  result  if  achieved  at  the  expense  of  quality.  The  weighting 
associated  with  the  respective  goals  is  presented  below. 

All  institutions 


Learning  Component: 

Accessibility  30 

Responsiveness  40 

Affordability  25 

Accountability  _5 

TOTAL  100 


For  the  research  component,  the  highest  weight  has  been  assigned  to  peer  review 
excellence.  The  ability  of  universities  to  contribute  to  economic  development  by 
transferring  new  knowledge  is  an  increasingly  prized  outcome.  Unfortunately,  the 
available  indicators  are  very  weak.  The  low  weighting  assigned  to  this  indicator  is  a 
product  of  its  statistical  quality  rather  than  the  desirability  of  the  outcome. 

Universities 

Research  Component: 


Excellence: 

Council  Monetary  Awards/Full-Time  Faculty  Member 

40 

Citation  Impact 

40 

Level  of  Community /Industry  Research 

_20 

TOTAL 

100 

Measuring  performance 

Two  performance  standards  for  each  indicator  are  proposed: 

(a)  achievement,  which  measures  an  institution’s  standing  relative  to  a standard  of 
excellence;  and 

(b)  improvement,  which  measures  an  institution’s  performance  against  its  own 
performance  from  the  previous  period. 

Achievement  is  weighted  more  heavily  than  year-to-year  improvement  (2:1)  to 
appropriately  reward  institutions  that  are  achieving  superior  performance  and  yet  offer 
encouragement  to  all  institutions  to  improve. 
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Differentiation 


For  most  indicators,  differentiation  based  on  size,  geography  or  mandate  is  not  necessary. 
Differentiation  is  important  for  the  cost-based  indicators  to  reflect  economies  of  scale  and 
program  mandate.  The  Cost/FTE  indicator  would  be  compared  within  program  clusters. 
Programs  would  be  grouped  according  to  subject  area  (for  example,  business,  arts  and 
science).  Appendix  III  lists  the  program  clusters  proposed.  An  institution’s  performance 
would  be  evaluated  in  its  four  program  clusters  with  the  highest  enrolment. 

As  well,  an  adjustment  factor  based  on  credit  enrolment  would  be  applied  to  the 
affordability  indicators  for  institutions  with  less  than  4,000  FTEs.  For  institutions  with  an 
enrolment  less  than  1,000  FTEs,  an  18  percent  adjustment  factor  would  be  applied.  The 
factor  would  decrease  to  a low  of  2.5  percent,  with  increasing  enrolment  (up  to  4,000 
FTE).  In  addition,  a multi-campus  adjustment  factor  would  be  applied  to  reflect  the  fixed 
costs  associated  with  decentralized  institutions.  In  most  cases,  institutions  have 
established  other  campuses  to  deliver  services  to  students  living  over  50  to  80  km  from 
the  main  or  home  campus.  It  is  proposed  that  the  affordability  indicators  for  institutions 
that  have  more  than  50  percent  of  their  FTE  enrolment  on  campuses  other  than  their 
home  campus  be  adjusted  by  7.5  percent,  a 5 percent  adjustment  would  be  applied  for 
enrolment  between  25  percent  and  49  percent  away  from  home  campus  and  a 2.5  percent 
adjustment  for  enrolment  between  10  percent  to  24  percent  away  from  home  campus. 
Both  of  the  adjustment  factors  noted  above  draw  from  formulae  used  in  Ontario. 

Benchmarking 

Ideally,  national  or  international  standards  of  superior  performance  would  be  used  as 
benchmarks.  Unfortunately,  such  data  is  not  available. 

Where  possible,  benchmarks  would  be  set  using  generally  accepted  standards  of 
excellence  (e.g.  the  satisfaction  indicators).  Where  we  are  forced  to  use  internal 
provincial  comparisons,  results  ranging  from  the  highest  level  of  achievement  to  the  first 
standard  deviation  would  receive  the  highest  rank.  Those  between  the  mean  and  one 
standard  deviation  above  the  mean  would  receive  the  second  highest  ranking. 

We  will  continue  to  push  for  PanCanadian  indicators  through  the  Canadian  Education 
Statistics  Council  to  promote  benchmarking  to  national  and  international  standards  of 
excellence. 

Corridor  concept 

The  performance  rating  calculation  uses  a “corridor”  concept.  All  institutions’  results 
within  a particular  range  of  performance  for  each  indicator  would  receive  the  same 
number  of  points.  Institutions  with  the  highest  rating  would  receive  a “1”  rating.  Those 
with  the  second  highest  rating  would  receive  a “2”  rating,  and  institutions  with  the  third 
highest  rating  would  receive  a “3”  rating.  Institutions  receiving  a “1”  rating  for  a 
particular  indicator  would  receive  the  maximum  number  of  points  available  for  that 


indicator.  Those  receiving  a “2”  rating  would  receive  about  two-thirds  of  the  maximum 
points,  while  those  with  a “3”  rating  would  receive  no  points. 

Example 

The  graphic  presentation  below  illustrates  the  corridor  principle  and  how  points 
would  be  awarded  for  results  within  the  three  different  ranges  described  above. 


Defining  Achievement  for  Affordability 


$12,000 

11,000 

10,000 

9.000 

8.000 

7.000 

6.000 

5.000 

4.000 

3.000 

2.000 

1,000 


Cost  per  FTE 


Institution  ►A  B 


Rating  ► 
Award  ► 


Determining  financial  awards 

Consistent  with  the  corridor  concept,  there  would  be  three  possible  awards  for  both  the 
learning  and  research  components  of  the  Performance  Envelope: 

Performance  Rating  Performance  Award 

1 3.25 

2 2.25 

3 1.50 


percent  (net  2.75) 
percent  (net  1.75) 
percent  (net  1 .0) 
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Proposed  performance  rating  calculation  for  1997-98 
and  1998-99:  Key  Performance  Indicators 


LEARNING  COMPONENT 

Group  1 Indicators  - Measures  quality  and  output 

ACCESSIBILITY: 


Maximum  Points  by  rating 
Points  1 2 3 


1. 

Credit  Full-Time  Equivalent  Enrolment: 

A)  Achievement 

16 

16 

11 

0 

B)  Improvement 

8 

8 

0 

0 

2. 

Completion  Rate: 

Improvement 

Sub-Total 

6 

6 

0 

0 

30 

RESPONSIVENESS: 

3. 

Student  Satisfaction  (Overall  Quality): 

A)  Achievement 

20 

20 

13 

0 

4. 

Employment  (Employed  in  Directly/Somewhat  Related  Jobs): 

A)  Achievement 

20 

20 

13 

0 

Sub-Total 

40 

SUB-TOTAL  GROUP  1 INDICATORS 

70 

Group  2 Indicators  - Measures  input  and  effectiveness: 

AFFORDABILITY: 

5. 

Cost  Per  Full-Time  Equivalent  Student: 

A)  Achievement 

13 

13 

9 

0 

B)  Improvement 

6 

6 

0 

0 

6. 

Instructional  Expenditures  As  Percent  of  Total  Operations  Expenditures: 

A)  Achievement 

B)  Improvement 

4 

4 

3 

0 

SUB-TOTAL  GROUP  2 INDICATORS 

2 

2 

3 

0 

25 

Group  3 Indicators  - Measures  accountability: 

ACCOUNTABILITY: 

7. 

Business  Plan  Complete/Submitted 

2 

2 

0 

0 

8. 

KPI  Templates  Complete/Submitted 

3 

3 

0 

0 

SUB-TOTAL  GROUP  3 INDICATORS 

5 

TOTAL  AVAILABLE  POINTS 

100 

RESEARCH  COMPONENT  (FOR  UNIVERSITIES) 

RESEARCH  EXCELLENCE: 

1.  Council  Monetary  Awards  per  Full-Time  Faculty  Member: 

A)  Achievement 

B)  Improvement 

2.  Citation  Impact: 

A)  Achievement 

B)  Improvement 

3.  Level  of  Community/Industry  Research  Funding: 

A)  Achievement 
TOTAL  AVAILABLE  POINTS 


25 

25 

17 

0 

15 

15 

0 

0 

25 

25 

17 

0 

15 

15 

0 

0 

20 

100 

20 

0 

0 
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Years  of  data  required  for  the 
1997-98  performance  rating  caculation 

LEARNING  COMPONENT 
Indicator 

1 . Credit  Full-Time  Equivalent  Enrolment 

2.  Completion  Rate 

3.  Student  Satisfaction  (Overall  Quality) 

4.  Employment  (Employment  in  Directly/Somewhat  Related  Jobs) 

5.  Cost  per  Full-Time  Equivalent  (FTE)  Student 

A.  Achievement 

B.  Improvement 

6.  Instructional  Expenditures  as  Percent  of  Total  Operations 
Expenditures 

A.  Achievement 

B.  Improvement 

7.  Business  Plan  Complete/Submitted 

8.  KPI  Templates  Complete/Submitted 

RESEARCH  EXCELLENCE  COMPONENT 

1 . Council  Monetary  Awards  per  Full-Time  Faculty  Member 

2.  Citation  Impact 

3.  Level  of  Community/Industry  Research  Funding 


Year(s)  Data  Used 

1994-95  and  1995-96 
1994-95  and  1995-96 
1994-95 
1994-95 

1994-95  and  1995-96 

1993- 94,  1994-95,  1995-96 

1994- 95  and  1995-96 

1993- 94,  1994-95,  1995-96 

For  December  1996 
For  December  1996 

1994- 95  and  1995-96 
1994-95  and  1995-96 
1994-95  and  1995-96 
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The  next  steps:  further  consultation 

We  intend  to  consult  with  each  institution  in  January,  as  well  as  with  the  Alberta  College- 
Institute  Faculty  Association,  the  Alberta  College  and  Technical  Institutes  Student 
Executive  Council,  and  the  Council  of  Alberta  University  Students.  The  office  of  each 
association  and  institution  president  will  be  contacted  shortly  to  schedule  the  consultation 
meetings. 

A final  document  describing  the  Performance  Envelope  will  be  released  in  early  March. 


Questions 

The  following  questions  will  be  considered  during  the  consultation  process: 

1 . Do  you  agree  that  the  Performance  Envelope  mechanism  should  be  relatively 
straightforward  and  easy  to  communicate? 

2.  Have  we  chosen  the  appropriate  indicators? 

If  not,  which  indicators  do  you  suggest? 

3.  Do  you  accept  the  weighting  that  has  been  assigned  ...by  goal  area?  ...by  indicator? 
...between  achievement  and  improvement? 

If  not,  what  weighting  do  you  suggest? 

4.  Do  you  accept  the  use  of  a corridor  concept  to  benchmark  for  each  indicator?  To 
determine  the  performance  rating  calculation  score? 

If  not,  what  do  you  suggest? 

5.  Do  you  agree  that  adjustments  should  be  made  to  cost  indicators  to  reflect  economies 
of  scale  and  decentralized  campuses?  Do  you  agree  with  the  proposed  method  of 
adjustment? 

If  not,  which  method  do  you  suggest? 

6.  Do  you  accept  the  means  used  to  set  rating  benchmarks? 

If  not,  what  means  do  you  suggest? 


Comments 

We  invite  you  to  comment  on  the  proposed  implementation  of  the  Performance  Envelope, 
using  the  questions  listed  above  as  your  guide.  Please  forward  your  comments  by 

January  24, 1997  to: 

System  Finance  and  Information 

Advanced  Education  and  Career  Development 

1 1th  floor  Commerce  Place 

10155  - 102  Street 

Edmonton,  Alberta  T5J  4L5. 


APPENDIX  I 
Indicator  Descriptions 


Instructional  component 

1 . Credit  full-time  equivalent 

This  indicator  measures  enrolment  in  credit  programs  using  FTE  enrolment  data.  The 
FTE  are  derived  from  the  Common  Information  System  (CIS)  definitions.  Enrolments 
supported  by  the  Access  Fund  and  Extension  Grants  will  be  deducted. 

2.  Completion  rate 

This  measures  the  percentage  of  students  who  have  completed  a program  of  studies 
compared  to  the  number  that  began  those  programs.  For  1997-98,  only  “parchment” 
programs  (programs  that  confer  a certificate,  diploma  or  degree),  would  be  considered. 

In  general,  it  is  expected  the  students  taking  a certificate  program  should  complete  it 
within  two  years.  Two-year  diploma  programs  should  be  completed  within  three  years 
and  a four  year  baccalaureate  program,  within  seven  years. 

3.  Student  satisfaction  (overall  quality) 

This  indicator  reports  how  students  have  evaluated  the  overall  quality  of  instruction  of 
their  respective  parchment  program.  The  1994-95  data  are  from  a follow-up  survey. 
Beginning  with  1996-97,  students  would  be  surveyed  upon  completion  of  the  program 
(exit  survey). 

4.  Employment  (employed  in  directly/somewhat  related  jobs) 

This  measures  the  extent  to  which  graduates  have  obtained  employment  within  six  to  nine 
months  (two  years  for  universities)  after  completing  a parchment  program.  This  is  a 
follow  up  survey  of  all  parchment  graduates. 

5.  Cost/full-time  equivalent  student 

This  measures  the  total  expenditures  per  FTE  student.  Expenditures  would  include  all  the 
direct  and  indirect  instructional  costs  including  appropriate  capital  amortization 
expenditures. 

For  the  improvement  portion  of  the  indicator,  total  institutional  credit  program 
expenditures  and  credit  FTE  numbers  would  be  used.  For  the  achievement  portion  of  the 
indicator,  cost/FTE  would  be  calculated  according  to  program  clusters  such  as  business, 
agriculture,  allied  health  and  technologies. 


15 


6.  Instructional  expenditures  as  a percent  of  total  operations 
expenditures 

The  intent  of  this  indicator  is  to  measure  the  portion  of  institutional  expenditure  allocated 
to  instruction. 

Instructional  expenditures  are  those  as  provided  under  the  heading  of  “Instruction 
Credit”,  “Instruction  Apprentice”  and  “Instruction  Non-Credit”  on  the  FIRS  Expenditure 
reports.  In  addition,  student  support  and  academic  support  expenditures  are  also 
included.  Total  Operations  Expenditures  is  derived  by  taking  the  “Total  Expenditure” 
and  subtracting  “Ancillary  Enterprise”  expenditures  and  for  the  universities,  in  addition, 
subtracting  expenditures  for  “Special  Purpose  & Trust”  and  “ Sponsored  Research”. 

7.  Business  plan  complete/submitted 

There  is  an  expectation  the  institutions  not  only  submit  annual  business  plans  on  time  but 
also  that  such  plans  are  complete.  This  would  include  consistencies  such  as  with  audited 
financial  statements,  CIS  enrolment  reports,  tuition  reports  are  accurate  with  respect  to 
the  Tuition  Fee  Policy,  and  staff  and  salary  settlement  data  are  accurate. 

8.  KPI  templates  complete/submitted/accurate 

The  same  principle  outlined  in  #7  above  would  apply  here. 


Research  component 

1.  Council  monetary  awards  per  full-time  faculty  member 

This  indicator  measures  the  ’productivity’  of  each  full-time  faculty  member  in  winning 
peer-adjudicated,  competitive  research  grants  from  the  three  federal  research  granting 
councils. 

2.  Citation  impact 

The  Citation  Impact  of  a publication  indicates  the  degree  to  which  a published  paper 
contributes  to  and  influences  its  discipline  or  field  of  study. 

3.  Level  of  community  or  industry  research  funding 

This  indicator  reveals  the  volume  of  funding  support  from  community  and  business 
sources. 
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Appendix  II 

The  performance  rating  calculation  in  practice 

To  illustrate  the  performance  rating  calculation,  KPI  pilot  data  and  CIS  enrolment  data 
has  been  used.  The  benchmarks  would  be  finalized  following  consultations  with 
institutions  and  following  receipt  of  December  KPI  data. 

1)  Credit  FTE 

A)  Achievement:  Each  institution’s  enrolment  (net  of  Access  Fund  and  Extension 

Grants)  would  be  converted  into  a system  share.  For  example,  institution  A may 
have  25  percent  of  system  enrolment  while  institution  B may  have  2 percent.  It  is 
possible  that  an  institution’s  enrolment  may  have  decreased  over  a particular 
period,  but  depending  on  relative  enrolment  changes  in  other  institutions  and  for 
the  system,  their  enrolment  share  may  actually  have  increased. 


Performance 

Point 

Share  Range 

Rating 

Scale 

Increase  in  Share 

1 

16 

Between  largest  increase  and  one 
standard  deviation  from  the  mean 

2 

11 

Mean  to  one  standard  deviation 

3 

0 

Below  the  mean 

B)  Improvement:  From  1989-90  to  1995-96  there  was  an  average  annual  increase  in 
enrolment  throughout  the  system  of  about  0.9  percent,  although  some  increases 
were  as  high  as  3 percent.  A benchmark  of  one  percent  could  be  established  (net 
of  Access  Fund  and  Extension  Grant  enrolments). 


Performance 

Point 

Percentage 

Rating 

Scale 

Increase 

1 

8 

Equal  or  greater  than  1 

2 

0 

Less  than  1 

2) 


Completion  rate 

Improvement:  Over  time,  it  is  expected  that  positive  increases  in  the  completion 
rate  would  take  place,  owing  to  a greater  emphasis  placed  on  successful  programs, 
linkage  with  the  business  sector  and  better  counseling  services  for  students.  A 
generally  accepted  economic  norm  is  that  productivity  should  improve  by  two 
percent  per  year. 


Performance  Point 

Rating  Scale 


1 6 
2 0 


Percentage 

Equal  or  greater  than  2 
Less  than  2 


3)  Student  satisfaction 

A)  Achievement:  Student  satisfaction  with  overall  quality  would  be  compared 

against  a standard  of  excellence.  It  is  expected  that  at  least  90  percent  of  the  total 
number  of  graduates  should  be  fully  or  somewhat  satisfied.  The  following  table 
shows  the  ranges  of  points  that  could  be  awarded. 


Performance 

Point 

Rating 

Scale 

Percentage 

1 

20 

Greater  than  90 

2 

13 

75  to  90 

3 

0 

Less  than  75 

4)  Employment 

A)  Achievement:  Employment  in  directly  related  or  somewhat  related  jobs) would  be 
measured  against  a standard  of  excellence.  It  is  expected  that  at  least  80  percent 
of  students  should  find  employ  in  directly  related  or  somewhat  related  jobs.  Each 
institution’s  employed  graduates  (from  survey  respondents)  that  fall  into  this 
category  would  be  expressed  as  a percentage  of  the  total  number  of  graduates. 
The  following  table  shows  the  ranges  of  points  that  could  be  awarded. 


Performance 

Point 

Rating 

Scale 

Percentage 

1 

20 

Greater  than  80 

2 

13 

70  to  80 

3 

0 

Less  than  70 

5)  Cost  per  full-time  equivalent  student 

For  purposes  of  this  example,  we  will  use  business  programs  and  1994-95  cost/FTE  data. 


A)  Achievement:  The  average  cost/FTE  for  business  programs  is  $6,800  per  FTE. 

The  range  of  costs  is  between  $5,000  and  $12,000  per  FTE.  The  majority  of 
institutions  are  within  a range  of  plus  or  minus  $1,200  from  the  mean.  Institutions 
with  a cost/FTE  over  $6,800  per  student  would  receive  0 points.  Those  with  a 
cost/FTE  between  $5,600  and  $6,800  would  receive  9 points,  while  those  below 
$5,600  per  FTE  would  receive  13  points.  The  following  table  shows  the  points 
for  each  range. 


Performance 

Rating  Scale  Cost  Per  FTE  Range 


13  Less  than  $5,600 

9 $5,600  to  $6,800 

0 Over  $6,800 


1 

2 

3 


B)  Improvement:  To  measure  improvement,  institutions  would  not  be  measured  on  a 
program-by-program  basis,  but  on  total  cost  and  total  credit  FTE  enrolment.  Total 
cost/FTE  would  be  measured  from  1993-94  to  1995-96. 

6)  Instructional  expenditures 

Data  has  been  derived  from  the  1993-94  and  1994-95  expenditure  templates. 

A)  Achievement:  The  average  instructional  expenditures  as  a proportion  of  total 
expenditures  was  61  percent  for  the  system.  There  was  a range  from  about  56 
percent  to  70  percent.  The  majority  of  institutions  were  in  a tight  range  of 
between  60  percent  and  65  percent.  Institutions  below  60  percent  would  receive  0 
points.  Those  between  60  percent  and  65  percent  would  receive  3 points,  while 
those  above  65  percent  would  receive  4 points.  The  following  table  shows  the 
points  that  could  be  awarded  for  each  range  of  instructional  expenditure. 


Performance  Point 

Rating  Scale 


Percentage  of 
Instructional  Expenditures 


2 

3 


4 

3 

0 


Greater  than  65 
60  to  65 
Less  than  60 
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Improvement:  On  average,  it  is  found  that  instructional  expenditures  as  a 
proportion  of  total  expenditures  decreased  from  1993-94  to  1994-95  by  2 percent 
across  the  system.  This  was  due  partially  to  the  reduction  in  compensation  that 
took  place  over  1994-95  and  into  1995-96.  Over  time  it  is  expected  that  positive 
increases  in  the  level  of  instructional  expenditures  as  a proportion  of  total 
expenditures  would  take  place.  For  this  indicator  a benchmark  of  “plus”  two 
percent  could  be  established. 


Performance  Point 

Rating  Scale 


Percentage 


1 

2 


2 Equal  or  greater  than  2 

0 Less  than  2 


Business  plan  complete/submitted 

All  institutions  would  be  awarded  2 or  0 points  depending  upon  whether  their 
business  plan,  requested  for  December  1996,  is  complete  and  has  been  submitted, 
including  all  templates. 

Key  performance  indicators  complete/submitted 

All  institutions  would  be  awarded  3 or  0 points  depending  upon  whether  their 
KPI  Templates,  requested  for  December  1996,  are  complete  and  submitted. 


Appendix  III 

Program  Clusters 


Subject 

Program  Cluster Area  Type 


Arts  and  Science 

AS,  BL,  HS, 
MP,  UC 

Education 

ED 

Business 

BU 

Social  Services/Law  SS 


Preparatory  and  Basic  Upgrading  AC,  CP,  ES, 

JL,  VP 


Visual/Fine/Performing  Arts  & FP,  CM 
Communications 


Agriculture/Engineering/ 

Computing 

AE,  AG,  IT, 
RR,  CE,  CR, 
EE,  EG,  ME, 
TR 

Services  and  Hospitality 

SH 

Medicine/Dentistry 

MD 

Program  Cluster 

Subject 

Area  Type 

Health 

HE 

Apprenticeship 


KPI  Program 
AP 


Types  of  Programs 

B.A.,  B.Sc.,  Faculte  St.  Jean,  Arts  and  Science  Native 
Studies,  Unclassified 

B.Ed.,  Physical  Education,  Early  Childhood 
Development,  Teacher  Assistant 

Commerce,  Business  Administration  Certificate  and 
Diploma,  Office  Administration  Certificate  and  Diploma, 
Human  Resource  Management,  Legal  Assistant,  Library 
and  Information 

Law,  Social  Welfare,  Child  and  Youth  Care,  Police  and 
Security,  Rehabilitation  Practitioner,  Criminal  Justice, 
Therapy  Assistant,  Social  Work  Diploma,  Legal  Assistant 

Academic  Upgrading,  Adult  Basic  Education, 
Transitional/Vocational,  College  Preparation  Job 
Readiness  Training,  Inmate  Education,  ESL 

Fine  Arts,  Visual  Communications,  Drawing,  Dance, 
Music,  Theatre  Arts,  Sculpture,  Communication  Arts, 
Broadcasting,  Graphic  Sign  Art,  Radio  and  Television, 
Journalism  Arts 

Agriculture,  Engineering,  Computer  Science, 
Environmental  Science,  Land  Resource  Management, 
Engineering  Technologies,  Information  Technologies, 
Transportation 

Hospitality  and  Tourism,  Baking,  Cooking,  Pastry  Chef, 
Meat  Cutting,  Travel  Consultant  Recreation  Facilities 
Operations 

Medicine,  Dentistry,  Dental  Hygiene 
Medical  Laboratory  Science,  Respiratory  Ther. 

Lab  and  X-Ray  Technician,  Dental  Assisting, 

Hearing  Aid  Services,  Ophthalmic  Dispensing 


Types  of  Programs 

Nursing,  Rehab.  Medicine,  Pharmacy, 

Gerontology,  Psychiatric  Nursing,  Mental  Health, 
Occupational  Health,  Pharmacy  Technician,  Allied 
Health  Technologies 

Carpenter,  Electrician,  Auto  Mechanic,  Plumber,  Welder, 
Parts  man,  etc. 
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